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We  conducted  in-depth  study  of  perfonnance  metrics  used  in  evaluating  intrusion  detection  systems.  We  define  Intrusion  Detection 
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I.  Project  Activities  and  Findings: 


We  conducted  in-depth  analysis  of  existing  metrics  used  in  evaluating  intrusion  detection 
systems,  and  illustrated  the  shortcomings  and  limitations  of  these  metrics.  We  define 
Intrusion  Detection  Capability  as  the  ratio  of  mutual  information  between  the  IDS  input 
and  output  to  the  entropy  of  the  input.  It  integrates  all  the  important  factors  into  a  single 
metric.  We  showed  that  this  new  metric  is  very  sensitive  to  IDS  operation  parameters. 
This  means  that  the  new  metric  can  be  used  to  guide  the  fine-tuning  of  IDS.  This  work 
was  published  in  the  ACM  Symposium  on  InformAtion,  Computer  and  Communications 
Security  (ASIACCS ’06). 

We  also  worked  on  an  information-theoretic  framework  for  analyzing  intrusion  detection 
systems.  We  defined  information-theoretic  metrics  to  measure  the  effectiveness  of  an  IDS 
in  terms  of  feature  representation  capability,  classification  information  loss  and  the 
overall  intrusion  detection  capability.  We  showed  that  intrusion  detection  capability  is 
equal  to  the  feature  representation  capability  minus  the  classification  information  loss. 
This  means  that  each  IDS  step/component  need  to  preserve  “information”  from  the  raw 
data,  e.g.,  feature  selection/construction  algorithms  need  to  be  improved  to  distinguish 
attack/normal  samples.  This  work  is  to  appear  in  the  1 1th  European  Symposium  on 
Research  in  Computer  Security  (ESORICS  2006). 

We  also  worked  on  IDS  alert  fusion  (i.e.,  how  to  effectively  use  multiple  IDSs).  It  is 
generally  believed  that  by  combining  several  diverse  intrusion  detectors  (i.e.,  forming  an 
IDS  ensemble),  we  may  achieve  better  perfonnance.  However,  there  has  been  very  little 
work  on  analyzing  the  effectiveness  of  an  IDS  ensemble.  We  studied  the  following 
problem:  how  to  make  a  good  fusion  decision  on  the  alerts  from  multiple  detectors  in 
order  to  improve  the  final  perfonnance.  We  proposed  a  decision-theoretic  alert  fusion 
technique  based  on  the  likelihood  ratio  test  (LRT).  We  evaluated  this  technique  using 
empirical  studies,  and  formally  analyzed  its  practical  interpretation  based  on  ROC  curve 
analysis.  Through  theoretical  reasoning  and  experiments  using  multiple  IDSs  on  several 
data  sets,  we  showed  that  our  technique  is  more  flexible  and  also  outperforms  other 
existing  fusion  techniques  such  as  AND,  OR,  majority  voting,  and  weighted  voting.  This 
work  was  published  in  the  ACM  Symposium  on  InformAtion,  Computer  and 
Communications  Security  (ASIACCS  ’08). 


